1. Field of the Invention
The present invention is a method and system to provide correspondences between the face camera tracks and behavior camera tracks in video surveillance or retail customer behavior analysis.
2. Background of the Invention
Increasingly, a large number of video cameras are being installed in public spaces such as airports, train stations, or public buildings for surveillance and public safety. Cameras installed in commercial spaces such as stores or shopping malls are also being used for monitoring or for video analytics for marketing purposes. These cameras are often connected to computer systems for automatic analysis, by tracking people and analyzing their behavior. Under some scenarios, the system needs to identify people or classify people into meaningful groups. It is sometimes necessary to make correspondences among people appearing in different camera views. In security applications, a person's behavior can be analyzed and recognized from video data feed by one camera view (potentially a far-range view), while the identity of the person can be recognized by a different (typically a close-range) view. In shopper behavior analysis applications, top-down camera views may provide the shopping behavior of customers while camera views capturing faces of customers may be used to identify the customer's demographic information.
Both of these applications typically track people within each camera view. When the collection of cameras cover the area to be monitored without too much gap, only the positions and the timestamps of the tracked person from different camera views can be utilized to make the correspondences. On the other hand, when the designated area is sparsely covered by cameras, additional information is required to make correspondences among people appearing in these sparsely positioned cameras to ensure the continuity of the tracks.
The present invention concerns the problem of multi-camera tracking of people, especially the issue of making correspondences between people appearing in face camera views and people appearing in behavior camera views. The correspondence problem is not trivial, because the appearance of a person in a face camera view is often significantly different from the appearance of the same person in behavior camera views, mostly due to different camera positions and angles. The face camera typically captures the facial image of people from slightly above the facial level, but mostly at angles not far from horizontal. The behavior camera orientation can be vastly different, from almost horizontal to vertical angles. On top of the viewpoint issue, the cameras can only observe the part of the body that is facing the camera. In extreme cases, one camera is able to capture only the front side of the person, while the other camera is able to capture only the rear side of the person. Therefore, the appearance changes of people due to camera angles and bodily pose can be dramatic, and is hard to model in a systematic manner; comparing the bodily appearances of people across the cameras becomes a major problem.
The present invention provides a system and method to make correspondences between person tracks in behavior cameras and person tracks in face cameras. As mentioned, it is necessary to utilize appearance information of people when there are significant gaps in camera coverage. As long as the system can determine whether two body images appearing in different views are instances of the same person, the correspondences can be made. However, it is nontrivial to solve, due to the drastic changes of appearance of body image across camera views. The present invention adopts a machine learning approach to solve the correspondence problem. The two candidate body images are paired into a single image; “the pairwise body image” is fed to a learning machine to be determined whether or not they are the same person. The machine is trained to estimate the likelihood of the two body images in the pair belonging to the same person.
The potentially very diverse body pose can still make it hard for the machine to verify the person's identity. In most camera placements, the camera position is static; therefore, the floor position of the person determines the body pose relative to the camera, assuming that the person is standing upright. This also applies to pan/tilt cameras, as long as the base of the camera is fixed. In one exemplary embodiment, the present invention divides the camera views, according to the distance from the camera axis, to handle each view using separate learning machines. The divided pose regions will have the form of area between concentric circles (or ellipses), and within each pose region the body image will go through the same amount of distortion. The angular position of the body will also determine the orientation of the body image. Therefore, the system will automatically recognize both the amount of distortion and the orientation based on the floor position. The body image is then corrected to have a standard pose (upright orientation and size), and is sent to the machine trained for that particular pose region.
There have been prior attempts for tracking people across multiple camera views. U.S. Pat. No. 6,950,123 of Martins (hereinafter Martins) disclosed a method and system for simultaneous tracking of multiple objects in a sequence of video frames captured by multiple cameras. The tracking may be accomplished by extracting a foreground elements from a background in a frame, segmenting objects from the foreground surface, tracking objects within the frame, globally tracking positions of objects over time across multiple frames, fusing track data of objects obtained from multiple cameras to infer object positions, and resolving conflicts to estimate the most likely object positions over time. Embodiments of the present invention improve substantially over existing trackers by including a technique for extraction of the region of interest that corresponds to a playing field, a technique for segmenting players from the field under varying illuminations, a template matching criteria that does not rely on specific shapes or color coherency of objects but on connected component properties, and techniques for reasoning about occlusions and consolidating tracking data from multiple cameras.
U.S. Pat. No. 6,987,885 of Gonzalez-Banos, et al. (hereinafter Gonzalez-Banos) disclosed systems, apparatuses, and methods that determine the number of people in a crowd using visual hull information. In one embodiment, an image sensor generates a conventional image of a crowd. A silhouette image is then determined based on the conventional image. The intersection of the silhouette image cone and a working volume is determined. The projection of the intersection onto a plane is determined. Planar projections from several image sensors are aggregated by intersecting them, forming a subdivision pattern. Polygons that are actually empty are identified and removed. Upper and lower bounds of the number of people in each polygon are determined and stored in a tree data structure. This tree is updated as time passes and new information is received from image sensors. The number of people in the crowd is equal to the lower bound of the root node of the tree.
U.S. Pat. No. 7,356,425 of Krahnstoever, et al. (hereinafter Krahnstoever) disclosed a method for calibrating a projective camera. The method includes acquiring information by detecting at least one object on a substantially flat ground plane within a field of view. A projective camera calibration is performed. A measurement uncertainty is considered to yield a plurality of camera parameters from the projective camera calibration.
U.S. Pat. Appl. Pub. No. 2004/0156530 of Brodsky, et al. (hereinafter Brodsky) disclosed a method and system that is configured to characterize regions of an environment by the likelihoods of transition of a target from each region to another. The likelihoods of transition between regions is preferably used in combination with conventional object-tracking algorithms to determine the likelihood that a newly-appearing object in a scene corresponds to a recently-disappeared target. The likelihoods of transition may be predefined based on the particular environment, or may be determined based on prior appearances and disappearances in the environment, or a combination of both. The likelihoods of transition may also vary as a function of the time of day, day of week, and other factors that may affect the likelihoods of transitions between regions in the particular surveillance environment.
U.S. Pat. Appl. Pub. No. 2005/0058321 of Buehler (hereinafter Buehler) disclosed a computerized method of image analysis that includes receiving first image data for a plurality of first video frames representing a first scene. Each first video frame is composed of a plurality of image regions, and a first object is present in an image region of at least one first video frame. Second image data is received for a plurality of second video frames representing a second scene. Each second video frame is composed of a plurality of image regions, and a second object is present in an image region of at least one second video frame. The method also includes determining a relationship between a first image region and a second image region based on a probabilistic correlation between occurrences of the first object being present in the first image region and occurrences of the second object being present in the second image region.
U.S. Pat. Appl. Pub. No. 2006/0028552 of Aggarwal, et al. (hereinafter Aggarwal) disclosed a unified approach, a fusion technique, a space-time constraint, a methodology, and system architecture. The unified approach is to fuse the outputs of monocular and stereo video trackers, RFID and localization systems, and biometric identification systems. The fusion technique is provided that is based on the transformation of the sensory information from heterogeneous sources into a common coordinate system with rigorous uncertainties analysis to account for various sensor noises and ambiguities. The space-time constraint is used to fuse different sensors using the location and velocity information. Advantages include the ability to continuously track multiple humans with their identities in a large area. The methodology is general so that other sensors can be incorporated into the system. The system architecture is provided for the underlying real-time processing of the sensors.
U.S. Pat. Appl. Pub. No. 2006/0285723 of Morellas, et al. (hereinafter Morellas) disclosed a system for tracking objects across an area having a network of cameras with overlapping and non-overlapping fields-of-view. The system may use a combination of color, shape, texture and/or multi-resolution histograms for object representation or target modeling for the tracking of an object from one camera to another. The system may include user and output interfacing.
In Martins, multi-person and multi-camera tracking is carried out by first motion-based segmentation, connected component grouping, and Sequential Monte Carlo multi-hypotheses tracking. When the tracker has observation gaps between camera views, it easily loses track of people. Gonzales-Banos mainly deals with the problem of resolving multiple views of the same person by constructing silhouette image cones and working volumes, and parsing the tree data structure constructed from intersections of image cones. Both Martins and Gonzales-Banos try to resolve the ambiguity among multiple instances of the same person under the assumption of overlapping cameras, while the present invention handles the correspondence problem between views that are non-overlapping. In Krahnstoever, the primarily concern is the problem of multi-camera calibration based on minimal information. The present invention adopts a similar method to achieve correspondence between overlapping camera views based on geometry, but also addresses the problem of corresponding non-overlapping camera views. Brodsky deals with the correspondence problem between targets appearing in and disappearing from camera views, based on transition probabilities between regions which are measured from color, size, shape, orientation, and motion of the tracked targets. Buehler makes correspondence between regions covered by different cameras and the correspondence between objects in different regions. Both Brodsky and Buehler aim to solve the multi-view correspondence problem by using prior transition information between regions. The present invention adopts a novel machine learning approach to solve the correspondence problems between any pair of object (or person) instances under any circumstances, without using any site-dependent prior information. It is the main task of Aggarwal to make correspondence between tracks from different sensors, including RFID, based on spatial, temporal, and appearance cues. The present invention assumes only visual information to solve the correspondence. In Morellas, the tracking of an object across multiple overlapping or non-overlapping cameras is handled by a particle filter-based tracker that utilizes color, shape, texture, and histograms. Under very challenging circumstances where the camera angles are extremely different, the appearance changes of people due to viewpoint are very difficult to model. Using a color, shape, or color histogram may fail to take into account this manner of nonlinear changes. The present invention deals with the appearance changes by learning it using a large number of data-pairs of body images, both matching and non-matching. One or more learning machine(s) is trained to determine whether or not a given pair of body images belong to the same person, which is one of the novel features of the present invention.